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intervention program based on the Reading Recovery/Whole Language 
theory. A group of 367 first-graie students who were identified as 
low-achieving was selected to participate in the 4-week program of 
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4-point scale) on a 26-item developmental checklist, with the 
difference between preprogram and postprogram means found to be 
statistically significant. This checklist was used when no common 
standard of comparison was available; the lack of a common standard 
made evaluating program success difficult. Levels of student 
attendance varied greatly, as did levels of parent involvement. Plans 
were made for continuation of the program in the 1994-95 school year, 
with recommendations for improvement centered on establishing a 
common standard for judging achievement and additional staff training 
and student retention efforts. (Contains 1 summary table, 5 figures, 
and 3 references.) (SLD) 



ED 391 818 

AUTHOR 

TITLE 

INSTITUTION 

PUB DATE 
NOTE 

PUB TYPE 

EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



ABSTRACT 



* * * vc vc vc vc * * y? y? * vc y? y? yc yc yc y? yc yc y? yc y? yc yc y? * it y? y? yc it it it it it it it it * yc it it * it it it it it * * yc it it it it it it * it * yc yc y? yc yc it it it it 

* Reproductions supplied by EDRS are the best that can be made 

* from the original document. 

it it it it it it it it it it yc it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it * it it 



o 

ERIC 



ED 391 818 






93.24 



THE ACCELERATING LITERACY PROGRAM: THE FIRST YEAR 19W-W 

Executive Summary 



Austin Ind^)ea<ient School District n * l V t* 

Office jf Research and Evaluation Author: Ralph J. Smtm 



Program Description 



Major Findings 



i lud2et implications 



The 1993-94 school year was the first year of the 
Accelerating Literacy Program (ALP) in AISD. The 
program was funded by a $223,599 Retention 
Reduction grant from the Texas Education Agency 
(TEA). The grant was used to train a group of AISD 
elementary educators in the methods of a short-term 
reading intervention program based on the Reading 
RecoveryAVholc Language theory. A group of 367 
first-grade students (1993-94 school year) who had 
been identified as low achieving and potentially "at 
risk" of dropping out in the future were selected to 
participate in a four-week program of accelerated 
reading instruction at 13 elementary campuses. All of 
these campuses were either on AISD*s "C" list, TEA’S 
list of schools that had "clearly unacceptable" TAAS 
results, or both. 

The program operates on the principle that, if low- 
achieving students arc offered short-term, intensive, 
and individualized remediation in an atmosphere which 
fosters and recognizes personal successes, they are 
more apt to succeed in their regular classroom 
environment. The program recognizes that the ability to 
read and write well is a requisite for success in all 
other academic areas. 

Four major objectives were originally defined for the 
program: 

1. All students who participate in the Accelerating 
Literacy Program will avoid the need to be retained; 

2. Students will have a stronger foundation for second 
grade, as shown by at least a four-level gain on the 
diagnostic running record in reading; 

3. Parents will become partners in the educational 
process as shown by their participation in workshops 
which stress home tcachLig strategies and their 
development of homemade materials and learning 
activities; and 

4. A school improvement planner will continue to 
expand literacy intervention efforts in these schools. 
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1. Programwidc, students increased their reading 

skills 0.47 levels (based on a four-point scale) 
on a 26-item developmental checklist. I 

difference in pro- and postprogram means was 
found to be statistically significant. However, 
the psychometric properties of the checklist arc 
unknown. This checklist was used as the 
primary indicator of progress when it was 
determined that no other common scale or 
criteria existed to measure programwide 
progress. Numerous problems with the running 
record, intended u an additional measure of 
reading progress, rendered it ineffective for 
purposes of program evaluation. (Pages 19-20) 

2. Nearly all ALP students were placed in the 
program based on recommendations made by 
their regular school year teachers. However, 
no systematic, common criteria were used on a 
pro^mwide basis to identify students. As a 
consequence, a number of students received 
very high ratings on the preprogram 
developmental checklist, indicating that some 
students may have been inappropriately placed 
in the program. (Page 22) 

3. No common set of reading achievement 
standards are being employed on a systemwide 
basis. Teachers at different campuses used 
various proficiency standards to judge their 
students* progress on running records. Lack of 
a common standard makes programwide 
success difficult to evaluate, and makes 
classroom-to-classroom or campus-to-campus 
comparisons impractical. (Pages 12, 21) 

4. Levels of student attendance varied 
considerably. Programwidc, students were in 
attendance only 86% of possible classroom 
hours. Many students who were nominally 
enrolled in the program (11.5% of total 
program enrollment) never reported for class; 
many more either formally withdrew or 
dropped out without explanation. (Page 16) 

5. Levels of parental involvement also varied. 
The parent training workshops at most schools 
were poorly attended. Parents were asked to 
maintain a parent-student reading activity bg 
over the four-week period. Examination of the 
logs revealed that a number of parents 
committed themselves to regular reading 
activities. However, a fubstantial number of 
parents did not submit logs, or submitted logs 
with very few entries. A majority of teachers 
reported that parenU of most of their studenU 
did not attend the workshops and did not keep 
a log of outside reading activities. 

(Pages 17-19) 

0 



Mandate: External funding agency 

Funding! Amount: $223,599 (1993-94) 

Funding Source: Division of 

Accelerated Instruction, Texas 
Education Agency 

Implications: Funds for continued 

support of the Accelerating Literacy 
Program in the 1994-95 school year 
have been procured. As this r^rt 
goes to print, it is not yet clear what 
form the program may take. It is 
anticipated that more elementary 
educators in AISD will receive ALP 
training during this school year, and 
that the program will be replicated as a 
four-week module during the summer of 
1995. Instruction may be extended to 
both kindergarten and second-grade 
students in the coming school year. 

Recommendations 

1 . The District should develop or adopt 
a common standard for measuring 
progress in reading achievement. 

2. Threshold proficiency levels should 
be defined by the same standards 
mentioned in the previous 
recommendation. Eligibili^ for the 
program should be determined by a 
two-part process. Students should 
first be identified by their regular 
teachers as potential participants. 
Then, those students should undergo 
a more formal diagnostic screening 
from an ALP-trained teacher before 
entering the program. 

3. All program staff should receive 
training in evaluation methodology. 
This instruction wiU lead to 
improved and more uniform record 
keeping, and ultimately, to more 
valid and reliable evaluation results. 

4. An infom.^1, nonbinding "contract" 
should be developed as a means to 
improve student attendance and 
parental involvement. 

5. Appointment of a permanent 
program coordinator similar to the 
school improvement planner 
discussed in the proposal to the 
funding agency would help ensure 
consistency within tk^ program. 
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PROGRAM EFFECTIVENESS SUMMARY 



ACCELERATING LITERACY PROGRAM 
1P/93-94 



Program 


Rating 


Ailocadon 

(Cost) 


Number 

of 

Students 

Served 


Cost Per 
Student 


Average 

Increa^i: 

Read^g 

Level* 

(Effect) 


Cost Per 
Stu^onito 
' Aitefn 
Ay 

indiHsaseih 

^ Level:';" \ . ; 

(Cost/Effecti 


Accelerating 

Literacy 

Program 


+ 


$223,599 


367 


$609 


0.47 


$1296 

Jj 



♦ Th» •v«rag* gain from pr^- to pcttprogfiiTL M dotorminod by • dtvtIofXTMntal choddi*t was 0.47 rMding b«Md on • 4-point acirfa whara a atudant'a 

damonttration of a akiN waa ratad aa 1 (Not Yat)« 2 (Oecaaionally}. 3 (Moat of tha Tima}, or 4 (Conaiatantty). 



Rating is expressed as contributing to any of the five AISD strategic objectives. 

+ Positive, needs to be maintained or expanded 
0 Not' significant, needs to be improved or modified 
Negative, needs major modification or replacement 
Blank Unknown 

Cost is the expense over the regular District per-student expenditure of about $4,000. 

0 No cost or minimal cost 

$ Indirect costs and overhead, but no separate budget 
$$ Some direct costs, but under $500 per student 

$$$ Major direct costs for teachers, staff, and/or equipment in the range of $500 per 
student or more 

ii 4 




93.24 



Accelerating Literacy Program 1993-94 



TABLE OF CONTENTS 

Executive Summary 

Program Effectiveness Summary 

List of Figures 

Conclusions and Recommendations 

Purpose of the Report 

Introduction 

Program Justification 

Funding the Program 

Selection of the Accelerating Literacy Program 

Population Served 

Outlook for Continuation of the Program 

Program Overview 

Evaluation Overview 

Revision of Program Objectives and Evaluation Questions . . 

Information Sources 

Indicators of Program Effectiveness 

Findings and Reconunendations 

Student Participation 

Parental Participation 

Progress in Reading Achievement 

Cost-Effect and Cost per Contact Hour 

Bibliography 



i 

ii 
iv 

I 

3 

4 

4 

5 

6 

7 

8 
9 

II 
11 
13 
13 
16 
16 
17 
19 

23 

24 




m 






93.24 



Accelerating Literacy Program 1993-94 



LIST OF nCURES 



Figure 1: AISD Elementary Schools Included on "C" List or on TEA’S List as "Clearly Unacceptable" . 4 



Figure 2; Budget for 1993-94 Accelerating Literacy Program 5 

Figure 3: ALP Student Demographic Summary 7 

Figure 4: Reported Enrollment and Attendance, by Campus, Summer 1994 16 

Figure 5: Mean Reported Total Miraites Spent in Parent-Student Reading Activities, by Campus 18 



b 



O 

ERIC 



IV 



93.24 



Accelerating Literacy Program 1993-94 



CONCLUSIONS AND RECOMMENDATIONS 



In its first year, the Accelerating Literacy Program (ALP) has shown promise of its ability to boost 
reading performance within the targeted "at risk" population of elementary students. Comparisons of 
pre- and postprogram performance ratings on a developmental checklist indicate positive, statistically 
significant gains in reading, writing, listening, and speaking skills on a programwide basis. The 
psychometric properties of this instrument are not known, however, and there is uncertainty about the 
checklist’s reliability. This finding supports the general consensus of program teachers and staff that the 
program was successful in its goal of raising students’ reading achievement and, it is hoped, will lower their 
risk of failure and dropping out later. 

The program realized somewhat less success in involving students’ parents in their children’s learning 
outside the classroom. However, this finding should be viewed with the knowledge that the program can 
only exercise limited influence on the parents’ commitment to their children’s education. There are a variety 
of other determinants that are beyond the control of the program which affect a parent’s willingness and 
ability to participate actively in his or her child’s schooling. However, parental involvement could prove to 
be quite critical in increasing students’ chances of academic success and reducing their chances of being at- 
risk in the future. Therefore, it is imperative that efforts are continued to raise the level of parental 
involvement. 

While the program can be judged to have been generally successful in its first summer (1994), several areas 
have been identified for improvement. The following are suggestions toward this end: 

• Formulate and adopt a common, grade level-appropriate set of reading standards to be used by all 
elementary schools in AISD. To insure that students are identified according to common criteria and 
standards, and to facilitate meaningful comparisons, a common set of formal diagnostic standards should 
be used districtwide. Adoption of a common set of standards will better enable District educators to 
measure progress in student reading achievement at all levels. Because the Accelerating Literacy 
Program has its basis in the Reading Recovery/Whole Language movement, and because AISD has a 
number of Reading Recovery-certified teachers, it seems logical that Reading Recovery levels might 
serve as the foundation for this set of standards. As more and more AISD teachers receive training in 
the Accelerating Literacy curriculum, such a set of standards will become logical to use districtwide. 

• Formulate and adopt a common method for identifying students who are likely to benefit from 
participation in the program. As indicated by the responses to the postprogram questionnaire completed 
by summer program teachers, nearly all students in the first-year program were identified according to 
the recommendation of their reguliir classroom teacher. However, most elementary teachers in AISD 
have not been tr ained in ALP met rods and many teachers may not be informed about the type of student 
the program is designed to serve. This situation seems to be ^parent from the fact that a number of 
students received ratings of "4" ("student consistently demonstrates this skill") on the 26-item preprogram 
developmental checklist. If these ratings are indeed reliable, then one is forced to ponder why these 
students were recommended for participation. The creation and adoption of formal and systematic 
diagnostic methods will help to ensure that only those students with the most to gain will be selected to 
participate in the program. 

• Consider the designation of a permanent coordinator for the Accelerating Literacy Program in AISD. A 
provision for a school improvement plaimer position was included in the original funding proposal, but 
was later deleted due to budgetary constraints. However, it is recommended that such a position be 
reconsidered for inclusion in the program. A permanent coordinator would help ensure a more consistent 
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program from year to year and from campus to campus. Preferably, the coordinator would be trained in 
Reading Recovery methods and, at the very least, will have receiv^ the ALP training. The coordinator 
might be responsible for organizing and leading a reading standards committee, conducting periodic ALP 
workshops to train new teachers (K-2), overseeing diagnostic evaluation of students before entry into the 
program, determining individual and program success t’ rough consistent and uniform record-keeping, 
improving efforts to elevate parental participation, and serving as an interface for other AISD persoimel, 
including the ORE evaluator, TEA staff, parents, and other interested individuals. 

• Continue to offer ALP training for AISD educators. I'raining workshops should be offered for new 
teachers as the ALP grows and develops. If a permanent program coordinator is hired, it would 
probably not be necessary to contract with an outside consultant each time a workshop is to be 
conducted. The coordinator might also conduct continuing education for those teachers already trained in 
the ALP agenda, if such a need is determined to exist. 

• Conduct training workshops on program evaluation techniques. It is important diat all teachers in the 
ALP understand the importance of the evaluation component to the educational process in general and to 
the ALP in particular. In view of the various record-keeping problems discuss^ elsewhere in this 
report, it is highly recommended that teachers receive some form of training in program evaluation. In 
order to evaluate program effectiveness reliably over time, it is essential that program documents be 
maintained in an accurate and consistent manner. The training might be offered in conjunction with the 
regular ALP workshops discussed above. ORE staff could jointly present this training with the program 
coordinator. 

• Continue and increase the use of ALP techniques in the regular school-year classroom. This 
recommendation will become more feasible as additional teachers receive the ALP training. By adopting 
the ALP ^proach to reading instruction, students should become familiar and comfortable with the 
reading materials, will be able to anticipate what is expeaed of them, and should continue to experience 
success in the classroom from year to year. With a consistent program of instruction in place, Aese 
students should be able to develop reading, writing, listening, and speaking skills concurrently. 

• Continue to develop strategies for encouraging parental participation in extracurricular reading activities 
with their children. Workshops should continue to be held during the regular school year, at various 
times during the day and evening to accommodate parents’ work schedules. Based on teachers’ 
comments, considerable exertion will probably be required to gain real commitment to attending these 
workshops. These efforts should include telq)hone calls to parents from teachers and counselors, as well 
as printed announcements sent home with students. The workshops should offer a variety of techniques 
to the parents for working with their children, arid should be relatively brief. Also, the Parent-Student 
Reading Activity Log should continue to be utilized during the regular instructional year. The design of 
the log should be reviewed and revised as needed. Teachers should use the log as an instrument for 
tracking parental participation, and as a means for communicating with parents. (The log should be 
returned to the teacher on a weekly basis. There is an area on the back cf rhe log for teachers and 
parents to exchange periodic comments about their children’s reading activity and progress.) Another 
idea for consideration is that of a ’’contract* agreement involving students, parents, and teachers. Such 
an arrangement would spell out the expectations of the program and the responsibilities of each party 
toward meeting the program goals. 

The findings on which the above recommendations are based are discussed in greater detail in the 

"Findings* section of this report. 
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PURPOSE OF THE REPORT 



This report is intended to serve as an initial basis for an ongoing evaluation of the Accelerating Literacy 
Program (ALP). The program was first introduced into the Austin Independent School District (AISD) in 
the spring term of the 1993-94 school year when a selected group of elementary teachers received training 
in a modified version of Reading Recovery. The first group of students (first graders in the 1993-94 school 
year) to participate in the program did so in the summer of 1994. The report is also intended to inform and 
guide decision makers in Ae District and in the Division of Accelerated Instruction of the Texas Education 
Agency. The report sununarizes program information, rates the effectiveness of the program, suggests 
possible modifications and improvements for the future, and, it is hoped, will serve as a formative guide 
and evaluation blueprint as the program progresses into its second year. 

In reality, genuine program effectiveness cannot truly be determined for at least several years; that is, the 
progress of those "at risk" students who chose to participate in the initial program will need to be tracked, 
assessed, and evaluated during their tenure in AISD schools and compared to that of students not at risk. 
Therefore, this report should be viewed from a formative rather than a summative perspective. The main 
objectives of the report are to comment on the effectiveness of the Accelerating Literacy Program in 
addressing the objectives set out in the original proposal, to draw some preliminary conclusions on both the 
short- and long-term prospects for the future, and to make recommendations for improving the efficacy of 
the program. 
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ACCELERATING LITERACY PROGRAM 
nNAL REPORT 
INTRODUCTION 



Program Justification 



The catalyst behind the 1993-94 Accelerating Literacy Program may be found within two official lists: 

1) the AISD "C" list (i.e., those four elementary schools with less than 25% of students passing all 
sections of the spring 1994 Texas Assessment of Academic Skills (TAAS)), and 2) TEA’s list of schools 
whose TAAS scores were "clearly unacceptable" (i.e., those nine schools with fewer than 20% of 
students passing all sections of the TAAS). All schools on these overlapping lists are Chapter 1 schools 
and serve a largely minority population. Those 13 schools are identified in Figure 1. With the 
overarching goal of removing the 13 schools from both of these lists, AISD has sought to boost student 
performance in these schools by identifying low-achieving, "at-risk" students for early and intensive 
intervention. Though these schools and students are identified on the basis of fourth-grade TAAS scores, 
it is a commonly held view among professional educators that those low-performing fourth graders have 
mostly been bdtind their peers since the beginning of their school careers, and that intervention is 
required in the lower grades in order to improve test scores in higher grades. 

FIGURE 1 

AISD ELEMENTARY SCHOOLS INCLUDED ON "C LIST OR ON 
TEA’S LIST AS "CLEARLY UNACCEPTABLE" 



CAMPUS 

NAME 


PERCBKTAGBOP 

srruoENT 

ENROLLMENT FAILING 
TO MEET MINIMUM 
EXPECTATIONS ON ALL 
PORTIONS OP TAAS 
(Spring 1^94)^ 


Allison 


56% 


Blackshear 


69% 


Blanton 


62% 


H Govalle 


57% 


1 Jordan 


79% 


\ Metz 


73% 


[ Norman 


72% 


Oak Springs 


59% 1 


Pecan Springs 


79% 1 


Ridgetop 


9% 


R Sims 


70% 


y Winn 


70% 


1 Wooten 


48% 1 



(* includcf all ttudenU not in special education.) 
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In most cases, students were identified for participation in the pilot program on the basis of 
recommendations by their regular school-year teachers and counselors and, sometimes, on the basis of test 
scores. The majority of these students are from minority populations and generally come from lower 
socioeconomic strata. As mentioned in the proposal to Ae fiinding agency, many of the students conte from 
backgrounds of poverty and from families with low levels of formal education, and therefore do not begin 
their school careers as well prepared to learn as other students. In the past, these students might have been 
retained in first grade. However, AISD has adopted the view in recent years that retention has more 
negative repercussions than benefits and that retention should only be utilized in the most severe cases. But, 
in the absence of intervening measures, these same students who otherwise might have been retained start 
their school careers at a considerable disadvantage and may be passed through the system lacking the 
foundation necessary for higher levels of learning. 

In the absence of intervening measures, these students will fall further and further behind their peers in 
classroom achievement over the course of years. As the gap widens, the likelihood of these students 
remaining in school and successfully completing their public school careers as high school graduates 
diminishes each year. That is, the students’ probability of becoming "at risk" of dropping out increases as a 
function of an ever-widening achievement gap. Most educators are in agreement that the key to reducing 
the dropout rate lies in identifying those students as early as possible and placing them in intensively focused 
intervention programs. 



Funding the Program 

In response to a request for proposals (RFP) issuea by the Division of Accelerated Instruction of the Texas 
Education Agency (TEA), AISD submitted a proposal for funding the Accelerating Literacy Program in 
December 1993. The RFP constituted an invitation to apply for grant monies to be used to develop 
innovative programs to reduce grade-level retention. AISD submitted an application seeking a grant for 
$298,132 to fund the program for the 1993-94 sdiool year. The {qiplication was approved by TEA and 
AISD received funds from the Retention Reduction Grants Program in the amount of $223,599 on April 27, 
1994. The first-year budget is detailed in Figure 2 below. 

nGUR£2 
BUDGET FOR 

1993-94 ACCELERATING LITERACY PROGRAM 



Ciass/C^jet^ 
1 Description 


Budget 


g Payroll CotU 


$126,342 


1 

Profestional & 
Contracted Seivicea 


16,165 


Supplies & Msterisls 


77,577 


Other Operating Expenses 


3,515 


Total Direct Costs 
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Because the funds were awarded late in the regular school year, a decision was made to structure the 
Accelerating Literacy Program as a half-day summer program (in anticipation of an extended school year in 
1994-95) rather than as an after-school or special program during the regular school year. With the 
exception of two campuses, all schools began a four-week program of half-day instruction on June 6, 1994 
with the anticipation of serving approximately 480 AISD students. 



Selection of the Accelerating Literacy Program 

Recognizing that the ability to read well is essential to success in all other subject areas, several 
administrators within the 13 program schools and the District central office researched programs designed to 
deliver concentrated reading instruction to the targeted group of "at risk" first graders. With an eye toward 
either an extended school day and/or school year, these educators, through the Region XIII Education 
Service Center, identified a workshop program offered by Educational Consultant Services (ECS) of 
Imperial Beach, California. ECS’s in-service course had grown out of experiments in reading intervention 
in the South Bay (CA) Union School District, a district demographically similar to AISD, and with similar 
difficulties in reading achievement within "at risk" populations. Furthermore, the mechanics of the program 
resembled those of certain intervention procedures used at AISD’s Ortega Elementary School, a school that 
at one time experienced similar problems but which has staged a significant turnaround in TAAS scores. 

Working with South Bay Union School District personnel, ECS created the Accelerating Literacy Program, 
a reading curriculum characterized by intensive reading instruction, with roots in the Reading 
Recovery/Whole Language Programs developed by New Zealand educator. Dr. Marie Clay. (According, to 
the training material provided by ECS, New Zealand has the highest literacy rate in the world.) The 
Accelerating literacy Program varies from the Reading Recovery model in that it is adapted for small class 
settings, usually sU to eight students, rather than one-to-one instruction. The Accelerating Literacy 
Program is charac prized by the following: 

• Early Intervention, 

• Short-Term, Intense Help, 

• Building on Strengths, 

• Focus on How-To, 

• Action Oriented, 

• Accelerated Progress, 

• Reading/Writing Connection, 

• Focus on Meaning, 

• Sound-Letter Relationships, 

• Flexibility, and 

• St<0 Development. 
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In April 1994, trainers from ECS conducted a three-day, 15-hour workshop on the Accelerating Literacy 
Program curriculum for AISD teachers and administrators, most of whom participated in the 1994 summer 
program. Release time was paid to participating personnel. Among the main features of the program were 
lesson planning, diagnosis of particular problems by charting running records, miscue analysis training, and 
information on books and materials. A second 15-hour workshop was conducted by a consultant from ECS 
on June 20-21, 1994 for a group of kindergarten and second-grade teachers who will be utilizing some of 
the ALP techniques in their regular classrooms during the 1994-95 school year. A brief questionnaire, 
developed by ORE, was given at the conclusion of the second workshop. Responses to the questionnaire 
reflected general satisfaction with and enthusiasm for the program. The only negative comments concerned 
the length of the instructional day, an indication that a few teachers felt the breadth of material to be 
covered was perh^s somewhat too ambitious for a two-day workshop. 

The strategy of using these techniques in sequential grades (K-2) is intended to help ensure continuity in 
student learning by offering students a variety of consistent techniques by offering low-achieving students an 
opportunity to experience successes built upon proven learning strategies, thus laying a more solid 
foundation for learning in all subjects. 

In considering possible solutions to the problem of low reading achievement, AISD educators also identified 
an important resource to supplement the ALP curriculum: parental participation and involvement in their 
children’s reading education. It is well established that close parental participation in the education of 
young children is a strong determinant of early success in school. As District educators planned the first 
year of the Accelerating Literacy Program, they began to identify methods for actively involving parents in 
the program. A series of summer workshops was planned to educate parents on strategies and activities for 
developing and furthering their children’s reading ^ucation at home. In addition, a Parent-Student Daily 
Reading Aaivity Log was developed by the evaluation consultant and sent home with all students on the 
first day of instruction in the hope of encouraging parents to partake in reading activities with their children. 

Population Served 

Student demographic information, taken from District recordr, is shown in Figure 3. 

FIGURES 

ALP STUDENT DEMOGRAPHIC SUMMARY 



1 ^ 


PERCENT 


1 Male 


48.4% 


1 Female 


51.6% 


1 ETHNICITY 


PERCENT 


Hispanic 


52.8% 


African American 


43.8% 


White (not of 
Hispanic origin) 


3.2% 


Other 


0.2% \ 
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Outlook for Continuation of the Program 

The summer 1994 program was designed to serve as a pilot project, with the idea of continuing instruction 
either in the regular school year or during the summer segment of the extended school year. As this report 
goes to print, it has been learned that funds to continue the ALP program in the 1994-95 school year have 
been procured, but it is not yet clear what form the program will take. Though it is anticipated that some 
of the techniques of the ALP program will be used in classrooms during the regular school year, the ALP 
will probably continue primarily as a summer program. 
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PROGRAM OVERVIEW 



The Accelerating Literacy Program (ALP) as developed by Evaluation Consultant Services (ECS) was 
chosen for implementation in the summer session because of its reported success in a comparable 
environment (i.e., the South Bay (CA) Union School District). The curriculum is flexible enough that it can 
be used as a stand-alone program (such as the summer program) or as an adjunct to regular school year 
reading instruction. According to information in the teacher training manual, the curriculum is considerate 
of different learning styles, cultures, and language. 

The thematic principle upon which the program operates is that of integrating reading, writing, listening and 
speaking activities into a whole language approach to instruction. The program seeks to "accelerate" the 
learning process by building on students’ successes. According to the Aeory of the program, when a 
teacher determines that a student has mastered text at a given level, the teacher should immediately 
challenge the student at a higher level; this is the principle underlying the name of the prog ram. According 
to the ALP instruction manual, this objective is pursued through the following steps: 

• Evaluating the reading process through miscue analysis; 

• Applying the four types of reading (discussed below) in whole language instruction; 

• Using effective questioning strategies in guided process reading; 

• Assessing and monitoring reading and writing behaviors; and 

• Using predicting/confirming activities and mapping techniques that clarify thinking. 

Teachers received training in four modes of reading instruction: 1) Read Aloud, 2) Shared Reading, 3) 
Guided Process Reading, and 4) Silent Sustained Reading. In addition, two types of writing were covered: 
1) Modeled Writing, and 2) Student Writing. The program also included generation of ideas in lesson 
pl ann ing, cuing systems, prediction and confirmation strategies, diagnosis of particular problems by charting 
running records, miscue analysis training, program organization and management, and information on 
procurement of suggested texts and materials. A workshop program outline may be found in the technical 
report. 

A few schools were fortunate enough to have at least one teacher who had received training in Reading 
Recovery, in which the Accelerating Literacy Program is grounded. However, most of the summer 
program teachers worked at the same campuses where they were employed during the regular school year, 
and many were quite familiar with the students in the summer program, their learning styles, and individual 
learning difficulties. Indeed, in many cases the teacher who recommended a student for the summer 
program was the same teacher in whose classroom the student spent his or her summer. 

While all teachers received a common training and had the same tools and techniques at their command, 
they were given considerable latitude in structuring their classroom programs. Most teachers spent the first 
few days of the program establishing baseline ratings (i.e., running records and developmental checklist 
ratings) for their students, though many were already familiar with their students’ strengths and weaknesses 
since they had been in their classrooms during the previous school year. Through early use of the 
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developmental checklist, running record, and miscue analyses (and given their familiarity with their 
students), teachers were able to identify areas of weakness in individual students’ reading abilities and to 
design a learning strategy for each student. 

Though there were varying degrees of coordination between classrooms in each school, the program allowed 
each teacher to structure a program appropriate to the needs of his/her students. Teachers were encouraged 
to integrate a variety of reading, writing, speaking, and listening activities as the whole language philosophy 
dictates, with a focus on helping students develop self-monitoring, self-correcting strategies in their 
approach to reading. 
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EVALUATION OVERVIEW 



AISD’s Office of Research and Evaluation (ORE) was assigned responsibility for the required evaluation 
component of the program, a condition of the grant. An evaluation consultant, reporting to ORE’s senior 
evaluator, was contracted to undertake all tasks related to the design and execution of the evaluation. The 
consultant’s major tasks as identified in the proposal were: 

• To refine the basic evaluation design and questions as set forth in the proposal; 

• To develop and refine measures of student progress and performance; 

• To provide training to teachers in evaluation and data collection processes; 

• To design and create a longitudinal database containing both demogrjq)hic and student performance 
information for the purpose of tracking reading achievement progress over time; 

• To serve as liaison to the Texas Education Agency in its study of the impact of retention reduction 
programs in improving student achievement and lowering retention rates; and 

• To author a formative evaluation r^>ort at the conclusion of the summer program. 

It should be noted that the structure of the program changed in several important ways after the proposal 
was submitted. The program was not actu^ly funded until April 27, 1994. Partly for this reason, the 
program was scaled back from the regular school year calendar to a summer program. The evaluation 
consultant was brought aboard May 18. Summer classroom instruction began Monday, June 6, allowing 
little time for the evaluation consultant to conduct teacher training on the evaluation and data collection 
processes. Most of the intervening period was spent familiarizing the consultant with the Accelerating 
Literacy Program, in planning the evaluation effort, and in designing and creating instruments and 
instructions for data collection. 

Revision of Program Objectives and Evaluation Questions 

In the proposal submitted to the funding agency, four major program objectives were defined: 

1. All students who participate in the Accelerating Literacy Program will avoid the need to be retained; 

2. Students will have a stronger foundation for second grade, as shown by at least a four-level gain on 
the diagnostic running record in reading; 

3. Parents will become partners in the educational process as shown by their participation in workshops 
which stress home teaching strategies and the development of homemade materials and learning 
activities; and 

4. A school improvement planner will continue to expand literacy intervention efforts in these schools. 
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Since the District in recent years has adopted promotion policies aimed at discouraging grade-level 
retention, the first objective is largely moot. In information gathered in interviews and questionnaires, 
virtually all summer program teachers and principals indicated that few, if any, of the students selected to 
participate in the summer program would have been held back in order to repeat their first-grade year. 

The school improvement planner position mentioned in the fourth objective and originally included in the 
proposal to TEA was deleted from the program strategy, at least for the summer program. Thus, no effort 
was directed toward evaluation of that goal, though a discussion of the future of that position is included 
elsewhere within this report. 

With the elimination of objectives one and four from the summer program, evaluation efforts were 
concentrated around the second and third objectives. The evaluation consultant’s approach to the second 
objective had to be somewhat revised when it was discovered that no common scale or standard of reading 
achievement was being employed on a districtwide basis. Some teachers in the program used re'^ding levels 
defined by the publisher whose texts they were using in the classroom. Among publishers, there is 
considerable variation in definitions of reading levels. Other schools, especially those with Reading 
Recovery-trained teachers, used Reading Recovery levels to determine student progress. Lack of a 
commonly used standard therefore presented a considerable obstacle when attempting to assess programwide 
gains in reading proficiency. To determine the impact of the reading intervention methods on program 
students and to gauge the extent of parental involvement in their childrens’ reading education, several 
evaluation instruments were either adopted from external sources with modification or else were created by 
the evaluation consultant. Those measurement instruments are explained in »he following section. 

In attempting to judge the effectiveness of the sununer program in meeting the goals outlined in the grant 
proposal and in the objectives of the Accelerating Literacy Program, in view of the circumstances discussed 
above, the basic set of evaluation questions was revised as shown below: 

1. Did the Accelerating Literacy Program result in an overall increase in student reading levels? If so, 
by what criteria and measures? Were reading levels raised sufficiently that students are at 
comparable levels of reading achievement with their nonprogram peers? Apart from reading 
achievement, did the program positively impact the students? Their parents? The school faculty? 

2. What was the cost-effectiveness of the program? Did the average incremental increase in reading 
levels justify the costs of the program? 

3. Did the program successfully involve the parents in the intervention effort? Did it improve parents’ 
willingness to participate in their children’s education? 

4. Will the program need to be continued during the regular school year in order for these students to 
be able to keep pace with their classmates? If so, how should the program be structured during the 
regular school year? 

5. How did program instruction vary from classroom to classroom? From school to school? Did the 
variation produce different results? 
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Information Sources 

Several sources for determining program success in strengthening students’ reading foundations were used: 

• The Accelerating Literacy Summer Program Developmental Checklist; 

• The Summary of Running Record; 

• The Accelerating Literacy Summer Program Class Attendance Record; 

• End of Summer Program Teacher and Principal Questionnaires; 

• Teachers’ anecdotal records; and 

• Informal interviews with program staff. 

Two measures were used in determining the success of the program in involving parents in their children’s 
reading instruction: 

• Parent-Student Reading Activity Log; and 

• Informal discussion with program staff. 

These measures are discussed in detail in the following section. 

Indicators of Program Effectiveness 

At the beginning of the summer program, all teachers were issued a packet of forms and instructions 
developed by the evaluation consultant. Though none of the students were mandated to attend this summer 
program, teachers were nevertheless instructed to keep a class attendance record. To measure progress in 
reading skills, teachers were asked to rate every student in the areas of reading, writing, listening, and 
speaking using a list of 26 learning indicators on a pre- and postprogram developmental checklist. A more 
detailed measure of reading progress was the running record. Teachers were asked to use the ruiming 
record with their students to identify consistent error patterns and to isolate particular difficulties. In a 
running record, teachers would copy a short text from a student reader onto a record sheet. The student 
was then asked to read from the re^er. The teacher checked each word identified correctly, would mark 
words identified erroneously, and also words which were initially misidentified but which the student was 
able to correct by him/herself. When the student finished, the teacher was asked to calculate an accuracy 
rate and self-correction rate for the student. Each teacher was directed to construct a minimum of two 
running records per student during the four-week period, and to videotape the charting of beginning and 
ending running records for at least one student. 

Teachers were supplier! with the following forms for recordkeeping and data collection; 

Gassroom attendance record forms were used as indicators of student participation. Teachers were 
instructed to record the number of hours each student was present for class each day. The attendance form 
was presented as a simple chart, and most teachers were able to use It to record attendance information 
reliably. 
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The pre- and postprogram developmental checklist was designed to assess learning progress. Teachers were 
asked to use a four-part scale in rating each child on writing (nine indicators), reading (nine indicators), and 
listening/speaking skills (eight indicators). For the purposes of analyzing differences in pre- and 
postprogram performance, the scale was converted from a text to numeric format, where: 

1 = Not Yet (The student is not yet demonstrating this skill.) 

2 = Occasionally (The student occasionally demonstrates this skill.) 

3 = Most of the Time (The student demonstrates this skill most of the time.) 

4 = Consistently (The student consistently dem»;nstrates this skill.) 

For those cases in which there was no information on which to base a rating, or if the teacher chose not to 
rate students on a particular skill, or where interpreting a rating was uncertain or ambiguous (e.g., the 
teacher checked more than one rating per skill area, or placed hei mark halfway between cells), the 
respective table cell was left blank. 

Teachers seemed to have few difficulties utilizing the developmental checklist. 'Ilie form is clearly and 
logically organized, simple to use, and allows for an easy understanding and comparability of summary 
performance data. (The checklist was also translated into Spanish at the request of teachers at one campus.) 
Ratings of 26 indicators (nine each in Reading and Writing, eight in Listening/Speaking) were used to 
measure change in reading development over the course of the program. Ratings for all skills within each 
category were equally weighted. Means were calculated for each skill and each category. 

The running record form was intended to be used as a more precise indicator of improvement in reading 
accuracy and self-correction abilities; it was to have been an important diagnostic tool for this program. 

The form is designed so that the teacher transfers text from short passages onto the form, then charts the 
student’s reading performance on a word-by-word basis. The teacher checks off each word correctly 
identified and pronounced by the student and marks and notes the type of each miscue. After the exercise is 
finished, the teacher records the total number of "running words" in the text and total miscues or errors 
committed by the student, then calculates the accuracy rate {Accuracy Rate = Number of Correct 
Words/Total Running Words). Teachers are also asked to calculate Ae student’s self-correction rate {Self- 
Correaed Errors/Total Errors) and to rate the text categorically (i.e.. Easy, Instructional, or Hard). 
Teachers were asked to chart at least two running records per student during the program, in order to 
facilitate analysis of program effectiveness. Except in those cases where students dropped out of the 
program early, most teachers complied with this request. Indeed, it was found in many cases that teachers 
charted four, six, eight, or more running records for their students, indicating a certain enthusiasm for the 
technique and a belief in the information which can be derived from it. 

The Parent-Student Reading Activity Log was designed as the primary instrument for determining the level 
of parental participation in reading-related activities with their children. Parents were asked to describe 
reading-related activities shared with their children (e.g., listening to their child read aloud, sharing in the 
identification of "environmental print" (e.g., signs), trips to the public library, etc.) outside the classroom 
and to record the amount of time spent each day in those activities. Most parents who chose to use the log 
seemed to have had little difficulty in doing so. 
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A secondary goal of the summer program was the creation of student peiformance portfolios. With 
increasing attention in educational circles to so-called "authentic" assessment in the last several years, it was 
felt that the establishment of a new program in such an early grade afforded the District an excellent 
opportunity to begin building student portfolios which can follow students through their careers in the public 
schools. Teachers in the ALP summer session were asked to construct student folders containing copies of 
all of the previously discussed documents, as well as the teacher’s anecdotal notes on classroom observation, 
student writing samples, and other records which the teacher deemed relevant to the program. 

During the last week of the program, teacher and principal program questionnaires were distributed. 
Teachers were asked only to identify themselves by their campus, but not by name, so relative anonymity 
was insured. Forty-five of the 55 teachers (82%) returned completed forms. A synopsis of teachers’ 
responses is included in the technical report, but the essence of the questionnaire was to determine how the 
teachers felt about the effectiveness of the program. 
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FINDINGS AND RECOMMENDATIONS 



Student Participation 
Finding 

An examination of class attendance logs revealed a variety of patterns and problems with student 
attendance at nearly every campus. Summary attendance data by campus is shown in Figure 4. Across 
the 13 campuses, students were present only 86% of possible hours, on average. Many students who were 
enrolled in the program never reported for class (48 of 415 students, or 1 1.5% of total program 
enrollment). There were a number of variations in attendance patterns, according to class attendance 
records. For example, some students enrolled in the program attended the first week, were absent the 
following week or two, then returned for the final few days of the program. In a few cases, students did 
not report to their campuses until the second or third week of the program. Many more students formally 
withdrew or else dropped out without notification before the end of the program, despite efforts by teachers 
and administrators to contact their parents. 



nGURE4 

REPORTED ENROLLMENT AND ATTENDANCE, 
BY CAMPUS, SUMMER 1994 



CAMPUS 


Number of 
Students 
BntoEed 


Nun^)er of 
Students 
Ri^rting In 
Pina Week 


Weighted Mean 
PetcenUges of 
Total Houji 
Attended* 


AUiaon 


26 


26 


69% 


Blackahear 


28 


28 


82% 


Blanton 


30 


29 


95% 


Govalle 


83 


53 


70% 


Jordan 


30 


30 


71% 


1 Metz 


38 


35 


88% 


1 Nonnan 


23 


22 


76% 


0 Oak Springs 


25 


21 


81% 


1 Pecan Springs 


28 


22 


73% 


RidgeCop 


17 


17 


78% 


Sims 


26 


24 


68% 


Wnn 


37 


36 


84% 


Wooten 


24 


24 


78% 


1 Program 
1 Total 


415 


367 


86% B 



* Me«n pcrcenUgea include thoee itudenti who reported for cUm in the ftrrt week, but later withdrew or dropped out. Percentagea do 
not include thoee ttudenta who were enrolled but never reported for claac. 
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Recommendation 

Because participation in the program is entirely voluntary, and because students were not retained for failing 
to attend, it is probably realistic to expect similar attendance patterns in the future. If continued as a 
summer program, it will be important to develop ideas for improving attendance further. Schools might 
give consideration to an informal "contract" arrangement between parents, students, and teachers which 
would spell out the intentions, goals, and expectations of the summer program, and the responsibilities of 
each party toward meeting those goals. Wording of the contract should include references to student 
attendance expectations and to parents’ commitment to attend the parent training workshops and to spend 
time with their children in reading-related activities (see discussion following). However, program staff will 
need to continue to make follow-up contact with parents to explain the importance of the program to their 
childrens’ academic future and the more immediate need for their children to attend. 



Parental Participation 
Finding 

Utilization of the Parent-Studrat Reading Activity Log was highly variable. Many parents were quite 
diligent in reporting daily reading activities; some of the logs reflect dozens of hours of shared activities. 
However, many portfolios contained no logs whatsoever. Several plausible explanations exist: 1) parents 
shared reading-related activities with the students and maintained daily entries but failed to return the log at 
the conclusion of the program; 2) parents shared reading-related activities with the students, but did not 
maintain a daily log; or 3) parents did not engage in activities with their children and therefore did not keep 
a log. While it is possible that some of the logs may have been maintained but misplaced or not returned at 
the end of the program, it seems evident from the number of missing logs that many parents either worked 
with their children but did not maintain a record of their activities, or did not engage in any extracurricular 
reading activities with their children. Twenty-six of the 42 teachers (62%) completing the postprogram 
questionnaire also answered "no" when asked whether parents of their students were regularly maintaining 
and returning the reading activity log. Figure 5 summarizes mean total minutes of shared reading activities 
for each campus in the program. 

For those parents who maintained one, the log apparently was easy to use since most entries were easily 
interpretable. A few parents seemed to have misunderstood the purpose of the log, recording all 
extracurricular activities in which the student participated, e.g., swimming, television viewing, etc. 
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FIGURES 

MEAN REPORTED TOTAL MINUTES 
SPENT IN PARENT-STUDENT 
READING ACTIVITIES, BY CAMPUS 



CAMPUS 


Number of 
Student 
Portfolios 
Containing Logs 
with Entries 


Mean Total* 
Minutes Spent 
in Parent- 
Student Reading 
Activities 


Allison 


9 


239 


Blackshear 


4 


309 


Blanton 


6 


cS 


Govallc 


0 


NR 


Jordan 


11 


330 


Metz 


25 


345 


Norman 


0 


NR 


Oak Springs 


1 


85 


Pecan Springs 


2 


313 


Ridgetop 


15 


237 


Sims 


1 


15 


Winn 


2 


55 


j Wooten 


8 


248 


I Program 
1 Total 


84 


269 



NR = No Record* 

* Program mean was calculated by weighting each achool’s mean total minute* by the reapcctive number of portfolios submitted by 
each school. Those weighted mean* were added, then divided by the total number of *tudent portfolio* for the program. 



The parent-training workshops were not as weli-attended as had been anticipated. Despite ^act that 
workshops were often offered at a variety of times (both day and evening) and dates, and despite e’^forts by 
program staff to contact parents, overall participation was poor. Although a few campuses reported good 
attendance at their workshops, they seem to have been exceptions to the general trend. Though workshop 
sign-in sheets were not submitted as part of the evaluation process, a majority of teachers (25 of 42; 59%) 
responding to the end-of-program questionnaire answered "no" when asked whether parents of most of their 
students had attended any workshop. A majority of those teachers responding to the questionnaire indicated 







18 



93.24 



Accelerating Literacy Program 1993-94 



that most parents of their students had not attended the parent training workshops (25 of 45; 56^ and that 
most parents had not maintained the Parent-Student Reading Activity Log (26 of 45; 58%). Though the 
majority of teachers reported low parental participation, most asserted an observable correlation between 
levels of parent-student interaction and increases in reading achievement (29 of 45; 64%). These responses 
generally corroborate the evaluation findings. 

Recommendation 

Though the program was successful in raising reading achievement levels (see discussion in the following 
section) despite uneven levels of parental participation, it will be important to continue efforts to involve 
parents in their children’s early education and to make them aware of their potential to make a critical 
difference in their children’s future. Provided that funds are available, program schools should consider 
offering periodic parent training workshops throughout the regular school year. Reasonable efforts should 
be made to accommodate the work schedules of busy parents; in order to involve the greatest possible 
number of parents (and to save the costs of duplicated efforts), workshops might be held at rotating sites in 
the evening as well as on the occasional Saturday. 

It will be important to gain the commitment of the parents to participate in the workshops. In addition to 
the "contract" agreement discussed above, teachers and counselors (and parent training specialists, if they 
are funded) will probably need to do extensive tel^hone work. Another suggestion for gaining commitment 
is to send notices and sign-up forms home with students. The workshop announcement would list the time, 
date, and location of each workshop. The sign-up form would request parents to indicate whether or not 
they planned to attend the workshop, and to return the form in either case. If parents are compelled to 
indicate their intentions, it seems more likely that they will pledge themselves to attending the workshops. 
Reminder notices could be sent home with students a day or two before the workshops are held. 

Though some revisions may be required, the Parent-Student Reading Activity Log should continue to be 
used to monitor parental involvement. (In fact, there is no reason to limit use of the log strictly to program 
students, or to restrict its use only to the summer session.) During the parent training workshops, parents 
should receive training on the purpose of and procedure for maintaining the log. It will be important that 
teachers review students’ logs on a regular basis, and make efforts to contact parents who are not utilizing 
the log. 



Progress in Reading Achievement 
Finding 

Analysis of the developmental checklist indicator ratings shows that, programwide, students participating 
in the summer Accelerating Literacy Program gained an average 0.47 levels (preprogram mean rating 
= 3.03; postprogram mean rating = 3.50) on their overall reading skills (based on a four-point scale, 
where a ”4” is the highest possible rating). The difference in pre- and posqirogram mean reading ratings 
was found to be statistically significant . Detailed information concerning calculation of the t statistics may 
be found in the technical rqwrt. 

This finding supports the belief asserted by teachers in the postprogram questionnaire that the program had 
had a positive effect on most of their students. It is important to note, however, that some teachers 
appa’'^ntly viewed the program more as a maintenance effort than as a means for "accelerating" the 
learning process. Others felt the program had helped boost reading achievement among most of their 
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students, but had not met the goal of bringing the students up to level with their nonprogram peers. A 
minority of teachers (16 of 42; 38%) responding to the questionnaire stated that they felt the program had 
been effective to the point that most of their students were performing at the same level as their nonprogram 
peers. Several teachers qualified their remarks, commenting that if the program dates had been extended, 
or had the instructional days been longer, that their students might have eventually caught up. 

In response to the question on the postprogram questionnaire, "Given the current pace, do you believe your 
students will be reding on the same level in the fall as those first graders who were not identified for 
participation in the Accelerating Literacy Program?" the teachers were nearly evenly divided {"Yes" = 16, 
"No" =18, "Other" = 8). Among those teachers responding "no," most indicated that they still felt the 
program had value. Many commented that if they had had more time to work with the students (i.e., a 
longer school day or more program days) that their students might have improved to the level of their 
nonprogram peers. Oihers stated that if some of the students were not performing at the same level as their 
peers, it was probably due more to the fact of sporadic attendance than that the program was ineffective. A 
few teachers commented that they viewed the program more as a "maintenance" measure; that is, they felt 
the program helped low-performing students sustain whatever reading level they had attained by the end of 
first grade, preventing them ttom backsliding over the summer months but not advancing their skills. 

Recommendation 

For those teachers who taught in the summer 1994 program and who will continue to do so in the future, it 
should be emphasized that the agenda of the program is not intended to serve as a mere "maintenance 
effort." Rather, as the name implies, it is to accelerate the development of the reading skills in at-risk 
students as swiMy as possible to bring those students to age-^propriate levels of reading competency. 
Likewise, it is important that this program philosophy is firmly imparted when new teachers undergo the 
ALP training. It is central to the philosophy of the program for teachers to challenge their students 
continually with increasingly difficult materials. 



Einding 

As discussed in the "Evaluation Overview," the second of four major program objectives stated in the RFP 
was: 



"Students will have a stronger foundation for second grade, as shown by at least a four-level gain on 
the diagnostic running record in reading." 

In the planning of the ALP program, it was assumed that the student diagnostic running record, besides 
being an important diagnostic tool for classroom use, would also serve as an important measurement 
instrument for judging the level of overall program success. Howevo', a number of serious problems 
developed with the use of this form. The problems were so numa'ous and varied that they rendered 
this record virtually useless for determining overall program gains in student reading achievement 
during the summer program. Perh:q>s the teachers did not receive thorough or uniform training in 
marking and calculating the running record, or perhaps a set of instructions should have been included with 
the form provided by the evaluation consultant. 

There was considerable variation in the methods by which the running records were kept. Some teachers 
used forms which were different from those provided with the packet of evaluation materials, and, in a few 
cases, more than one type of form (other than the one provided at the beginning of the program) was used 
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to evaluate a single student. Often, teachers failed to mark records in a manner interpretable to the 
evaluation consultant. Quite frequently, running words, errors, accuracy ratios, and seif-correction rates 
were either partially or completely absent, or calculated incorrectly. Sometimes, it was clear that the 
teacher had charted the text, but had neglected to follow through with calculations of summary rates. In 
many cases, it was evident that the formulas required to calculate accuracy and self-correction rates were 
used incorrectly. Another common problem was that some teachers neglected to note the date that records 
were taken, thus making it practically impossible to conq>are early performances to later ones. The 
difficulty level of the text was sometimes not indicated. It was especially puzzling in some cases that 
teachers often submitted an initial running record showing a high accuracy rate on a text rated as "easy," 
followed by a final record with a high accuracy rate on a text of equal difficulty. By not challenging the 
student with a higher level of text, the teacher forced the student to work under a very low performance 
ceiling; that is, unless the student was moved to a higher level of difficulty, there was little room to 
demonstrate improvement (i.e., restriction of range). It is especially necessary to allow for ample "elbow 
room" when only two ruiming records are included in the portfolio, ’ihe converse is also true. When a 
running record reflects difficulty with text at a certain levd, the next running record should probably be 
taken at a lower level of difficulty. Some teachers seemed to have a clear understanding of the ^plication 
of this conc^t; however, many more did not. The reliability of the data taken from the running records is 
suspect to the point that it was determined that those records were of little use in gauging gains on a 
student, classroom, school, or program level. 

Recommendation 

The running record, when employed correctly, can constitute a valuable source of data both as an indicator 
of individual student pmgress as well as for program evaluation. Efforts to use it as part of an ongoing 
evaluation process should net be abandoned simply because of the difficulties encountered in the pilot phase 
of the program. However, it is suggested that teachers should receive further instruction in the use of the 
running record and on the importance of producing uniform, reliable, comparable, and comprehensible 
records. 

In the course of gathering information for the evaluation, it was discovered that no common set of reading 
proficiency standards was being employed on a programwide basis. Most teachers reported that they were 
using reading levels as defined by the publisher whose texts they were using in their summer class, and 
some rqported using texts from nnore foan one publisher. A few teachers also rq>orted using Reading 
Recovery-based levels. 

Such a projusion of differently defined reading levels makss any attempt to compare student progress across 
classrooms or schools both ^ffiadt and unreliable. Though individual teachers are probably able to assess 
each student’s progress quite accurately using a chosen scale, the lack of a uniform standard makes the 
assessment of programwide progress a difficult task. 

Since the Accelerating Literacy Program is grounded in Reading Recovery methods, it seems ^propriate 
that Reading Recovery levels ^ used as a common standard on a programwide basis. Several campuses 
were fortunate enough to have certified Reading Recovery teachers in the summer program. It is suggested 
that all teachers participating in the ne?t summer program receive instruction on Reading Recovery levels 
before the be ginning of the program, using one or more of these teachers as a presenter. If the ALP 
workshop is held again before the next summer program, the program agenda should be amended to include 
instruction on Reading Recovery levels. 
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Depending on the budget for next year’s program, either a Reading Recovery-trained teacher or a program 
coordinator (who might be the same person) could be assigned as a "roving observer" who would assist 
other teachers in charting running records using Reading Recovery techniques. Such a measure would tend 
to insure that student reading achievement is meaningfully comparable across campuses. 

Finding 

In the same vein as the finding discussed above, it was also discovered that students were not placed in the 
program by any common, systematic criteria. According to the majority of responses to the teacher 
questionnaire, nearly all students were elected to the program by teacher reconunendation. However, few 
of the respondents indicated the basis for making their recommendations, though several mentioned that they 
did so based on test results; however, none of those teachers named a specific test. Possibly as a result, it 
seems that a number of students may have been enrolled unnecessarily, if the student developmental 
checklist is a reliable, valid, age-appropriate instrument. A review of the completed checklists at the 
conclusion of the program indicates that a large number of students had received high ratings (i.e., "3’s" 
and "4’s") in most or all skills categories at the beginning of the program. Such a finding suggests that 
these may not be the students for whom the program was intended, or else that the checklist is not an 
adequate instrument for the task. 

RftCOTnmftntlatinn 

On the assumption that the skills categories on the developmental checklist are valid, that ratings of 
individual pupils are reliable, and that the checklist is an age-appropriate measure, it is suggested that an 
additional measure be used to screen students before accqitance into the program. Because some of the 
teachers making recommendations may not have received the ALP training and, therefore, may not 
understand which students the program is designed to serve, it is recommended that all students who are 
initially nominated for the program receive a preprogram diagnostic screening from a program-trained or 
Reading Recovery-trained teacher. 

For this suggestion to work effectively, however, there must be consensus on the level of need for 
placement of students in the program. Therefore, it is recommended that a committee be formed, composed 
of Reading Recovery teachers, teacher representatives from program campuses, and administrators, and, 
possibly, a permanent program coordinator, whose tasks would include, but would not be limited to: 

1) Defining threshold proficiency levels for entry into the program; 

2) Defining uniform reading levels for use across the program; 

3) Modifying or creating measurement instruments, such as the running record form, which can 
properly measure reading performance levels; 

4) Working with the evaluator to determine periodically the direction and progress of the program; and 

5) Developing and conducting continuing education activities for program personnel and District 
educators. 
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Cost-Effect and Cost per Contact Hour 
Finding 

One method for evaluating the effectiveness of the program is to consider the cost at which any net gain (or 
loss) is achieved. Much like unit pricing in a supermarket aisle, a cost-effect ratio informs us not only of 
the effectiveness of the program, but also of its efficiency. The cost-effect ratio is simply the calculation of 
total program cost divided by the mean gain (or mean loss) in reading achievement level, presuming that the 
gain is attributable to the program. In general, it is necessary to establish that the effect of a special 
program on student achievement is above and beyond the effect resulting from the students’ regular 
instructional program, as well as from other factors such as educational experiences outside school and 
normal maturation. Because the ALP was a summer program, it will be assumed for the purposes of this 
calculation that the observed gains in reading achievement were the result of the ALP and not of other 
instructional influences. The cost-effect ratio is expressed as: 

Cost-Effect = Total program budget = $223.599 = $47,574 

Mean gain/loss in reading achievement levels 0.47 

Another measure of program economy is to consider the number of students served and the amount of 
classroom contact time created per budget dollar. By dividing total program costs (i.e., the program budget 
or the one-year grant amount) by the number of total student-hours recorded by teachers (i.e., the sum total 
of classroom time for all students in the program, based on a four-hour instructional day), cost per contact 
hour can be derived. Thus, cost per contact hour for the 1993-94 summer program was calculated as: 

Cost per contact hour = Total program budg et = $223.599 = $9.50 per contact hour 

Total contact hours 23,520 



Recommendation 

Cost per contact hour can be reduced in two ways: 1) by reducing the program budget while maintaining 
total contact hours; or 2) by improving attendance and lowering the program dropout rate if the budget 
remains a constant. Assuming that the first option is the least desirable, emphasis should be placed on 
increasing total student-hoiurs by implementing the measures discussed in earlier sections of this report. 

Likewise, there are basically two methods for improving the cost-effect ratio. The first is to reduce the 
budget (cost) while maintaining improvements in reading achievement (effect). The other option is to hold 
costs steady while raising the rate of improvement in reading achievement. Again, it is assumed that the 
first option is undesirable. Therefore, the default option is to continue to improve the achievement gain. 
Perhaps the most immediate way to raise this effect is to improve ways of identifying the students who will 
benefit the most from the program. As discussed in an earlier section, there were a number of students who 
received high ratings in most of the skills areas at the beginning of the program. These students had little 
room for improved ratings; therefore, the possibilities for a large effect are limited for those students. As 
more students working under a low ceiling are placed in the program, the total effect for the program 
diminishes proportionally. By removing those students who are least likely to benefit from the program, the 
possibility for a larger effect is increased. Reducing enrollment and class size, of course, will idlow 
program teachers more time to spend with those students who are most in need of the benefits the program 
can deliver. (However, reducing student enrollment will have the concurrent effect of reducing the number 
of student contact hours, thereby raising the cost-per-contact-hour ratio.) 
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